{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ICESat-2 Hackweek 2020: Machine Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Instructors: Yara Mohajerani ([ymohajer@uci.edu](mailto:ymohajer@uci.edu)) and Shane Grigsby ([shane.grigsby@colorado.edu](mailto:shane.grigsby@colorado.edu))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial we will explore the basics of machine learning with an emphasize on neural networks and applications in altimetry. Part 1 will offer a brief introduction to theory and techniques in implementing neural networks, and Part 2 will focus on altimetry applications\n",
    "\n",
    "## Part 1\n",
    "Neural networks use a series of nonlinear transformations with adjustable (trainable) parameters to approximate an input field into a desired output.\n",
    "\n",
    "Each 'cell' or unit of a network has an associated weight $w$ and bias $b$, and an activation function $f(z)$ for applying a nonlinear transformation such that the output is\n",
    "\n",
    "$f(w.x + b)$\n",
    "\n",
    "for input $x$.\n",
    "\n",
    "Some examples of activation functions are \n",
    "Sigmoid:\n",
    "\n",
    "\n",
    "$f(z) = \\frac{1}{1+e^{-z}}$\n",
    "\n",
    "Rectified Linear Unit (ReLU):\n",
    "\n",
    "$f(z) = \\max(0,z)$\n",
    "\n",
    "And many others, which is beyond the scope of this brief tutorial. There are many excellent resources on the choice activation functions (include some introductory reads?)\n",
    "\n",
    "\n",
    "A neural network contains many layers of nodes to accomplish more involved transformations. Note that each unit only has one adjustable bias $b$, but each precedening connected note has a weight $w$ associated with it. All the weighted inputs are summed such that the output is\n",
    "\n",
    "$f(b+\\sum_{i} w_ix_i)$\n",
    "\n",
    "\n",
    "![Neural network generic example fromwww.astroml.org](https://www.astroml.org/_images/fig_neural_network_1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Packages like PyTorch and TensforFlow provide the tools to contruct neural networks in Python. However, here we focus on Keras, a higher-level package which makes it easier to contruct a network.\n",
    "\n",
    "Here as an example we will contruct a simple model for the quintessential machine learning example of identifying handwritten digits (MNIST dataset)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1) Install required packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting keras\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ad/fd/6bfe87920d7f4fd475acd28500a42482b6b84479832bdc0fe9e589a60ceb/Keras-2.3.1-py2.py3-none-any.whl (377kB)\n",
      "\u001b[K     |████████████████████████████████| 378kB 1.8MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: pyyaml in /Users/yaramohajerani/opt/anaconda3/lib/python3.7/site-packages (from keras) (5.1.2)\n",
      "Collecting keras-applications>=1.0.6 (from keras)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/71/e3/19762fdfc62877ae9102edf6342d71b28fbfd9dea3d2f96a882ce099b03f/Keras_Applications-1.0.8-py3-none-any.whl (50kB)\n",
      "\u001b[K     |████████████████████████████████| 51kB 16.9MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: scipy>=0.14 in /Users/yaramohajerani/opt/anaconda3/lib/python3.7/site-packages (from keras) (1.3.1)\n",
      "Requirement already satisfied: six>=1.9.0 in /Users/yaramohajerani/opt/anaconda3/lib/python3.7/site-packages (from keras) (1.12.0)\n",
      "Requirement already satisfied: h5py in /Users/yaramohajerani/opt/anaconda3/lib/python3.7/site-packages (from keras) (2.9.0)\n",
      "Requirement already satisfied: numpy>=1.9.1 in /Users/yaramohajerani/opt/anaconda3/lib/python3.7/site-packages (from keras) (1.17.2)\n",
      "Collecting keras-preprocessing>=1.0.5 (from keras)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/28/6a/8c1f62c37212d9fc441a7e26736df51ce6f0e38455816445471f10da4f0a/Keras_Preprocessing-1.1.0-py2.py3-none-any.whl (41kB)\n",
      "\u001b[K     |████████████████████████████████| 51kB 16.9MB/s eta 0:00:01\n",
      "\u001b[?25hInstalling collected packages: keras-applications, keras-preprocessing, keras\n",
      "Successfully installed keras-2.3.1 keras-applications-1.0.8 keras-preprocessing-1.1.0\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "pip install keras"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting tensorflow\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/35/55/a0dbd642e68e68f3e309d1413abdc0a7aa7e1534c79c0fc2501defb864ac/tensorflow-2.1.0-cp37-cp37m-macosx_10_11_x86_64.whl (120.8MB)\n",
      "\u001b[K     |████████████████████████████████| 120.8MB 342kB/s eta 0:00:01\n",
      "\u001b[?25hCollecting scipy==1.4.1; python_version >= \"3\" (from tensorflow)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/85/7a/ae480be23b768910a9327c33517ced4623ba88dc035f9ce0206657c353a9/scipy-1.4.1-cp37-cp37m-macosx_10_6_intel.whl (28.4MB)\n",
      "\u001b[K     |████████████████████████████████| 28.4MB 457kB/s eta 0:00:011\n",
      "\u001b[?25hRequirement already satisfied: keras-preprocessing>=1.1.0 in /Users/yaramohajerani/opt/anaconda3/lib/python3.7/site-packages (from tensorflow) (1.1.0)\n",
      "Collecting tensorboard<2.2.0,>=2.1.0 (from tensorflow)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d9/41/bbf49b61370e4f4d245d4c6051dfb6db80cec672605c91b1652ac8cc3d38/tensorboard-2.1.1-py3-none-any.whl (3.8MB)\n",
      "\u001b[K     |████████████████████████████████| 3.9MB 28.1MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting termcolor>=1.1.0 (from tensorflow)\n",
      "  Downloading https://files.pythonhosted.org/packages/8a/48/a76be51647d0eb9f10e2a4511bf3ffb8cc1e6b14e9e4fab46173aa79f981/termcolor-1.1.0.tar.gz\n",
      "Requirement already satisfied: wrapt>=1.11.1 in /Users/yaramohajerani/opt/anaconda3/lib/python3.7/site-packages (from tensorflow) (1.11.2)\n",
      "Collecting gast==0.2.2 (from tensorflow)\n",
      "  Downloading https://files.pythonhosted.org/packages/4e/35/11749bf99b2d4e3cceb4d55ca22590b0d7c2c62b9de38ac4a4a7f4687421/gast-0.2.2.tar.gz\n",
      "Collecting absl-py>=0.7.0 (from tensorflow)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/1a/53/9243c600e047bd4c3df9e69cfabc1e8004a82cac2e0c484580a78a94ba2a/absl-py-0.9.0.tar.gz (104kB)\n",
      "\u001b[K     |████████████████████████████████| 112kB 49.5MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting protobuf>=3.8.0 (from tensorflow)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/4c/25/c057a298635d08d087a20f51ff4287d821814208ebb045d84ea65535b3e3/protobuf-3.11.3-cp37-cp37m-macosx_10_9_x86_64.whl (1.3MB)\n",
      "\u001b[K     |████████████████████████████████| 1.3MB 37.9MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: wheel>=0.26; python_version >= \"3\" in /Users/yaramohajerani/opt/anaconda3/lib/python3.7/site-packages (from tensorflow) (0.33.6)\n",
      "Collecting tensorflow-estimator<2.2.0,>=2.1.0rc0 (from tensorflow)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/18/90/b77c328a1304437ab1310b463e533fa7689f4bfc41549593056d812fab8e/tensorflow_estimator-2.1.0-py2.py3-none-any.whl (448kB)\n",
      "\u001b[K     |████████████████████████████████| 450kB 52.0MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: six>=1.12.0 in /Users/yaramohajerani/opt/anaconda3/lib/python3.7/site-packages (from tensorflow) (1.12.0)\n",
      "Requirement already satisfied: keras-applications>=1.0.8 in /Users/yaramohajerani/opt/anaconda3/lib/python3.7/site-packages (from tensorflow) (1.0.8)\n",
      "Collecting opt-einsum>=2.3.2 (from tensorflow)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/b2/49/2233e63052d5686c72131b579837ddfb98ba9dd0b92bb91efcb441ada8ce/opt_einsum-3.2.0-py3-none-any.whl (63kB)\n",
      "\u001b[K     |████████████████████████████████| 71kB 17.1MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting grpcio>=1.8.6 (from tensorflow)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/35/f5/9dce584256802e9d26f51e689bf42b365442c6c52f1af8a44c86ad62359a/grpcio-1.27.2-cp37-cp37m-macosx_10_9_x86_64.whl (2.5MB)\n",
      "\u001b[K     |████████████████████████████████| 2.5MB 29.0MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: numpy<2.0,>=1.16.0 in /Users/yaramohajerani/opt/anaconda3/lib/python3.7/site-packages (from tensorflow) (1.17.2)\n",
      "Collecting astor>=0.6.0 (from tensorflow)\n",
      "  Downloading https://files.pythonhosted.org/packages/c3/88/97eef84f48fa04fbd6750e62dcceafba6c63c81b7ac1420856c8dcc0a3f9/astor-0.8.1-py2.py3-none-any.whl\n",
      "Collecting google-pasta>=0.1.6 (from tensorflow)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/a3/de/c648ef6835192e6e2cc03f40b19eeda4382c49b5bafb43d88b931c4c74ac/google_pasta-0.2.0-py3-none-any.whl (57kB)\n",
      "\u001b[K     |████████████████████████████████| 61kB 5.4MB/s eta 0:00:011\n",
      "\u001b[?25hRequirement already satisfied: setuptools>=41.0.0 in /Users/yaramohajerani/opt/anaconda3/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow) (41.4.0)\n",
      "Requirement already satisfied: werkzeug>=0.11.15 in /Users/yaramohajerani/opt/anaconda3/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow) (0.16.0)\n",
      "Collecting markdown>=2.6.8 (from tensorboard<2.2.0,>=2.1.0->tensorflow)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ab/c4/ba46d44855e6eb1770a12edace5a165a0c6de13349f592b9036257f3c3d3/Markdown-3.2.1-py2.py3-none-any.whl (88kB)\n",
      "\u001b[K     |████████████████████████████████| 92kB 25.6MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting google-auth<2,>=1.6.3 (from tensorboard<2.2.0,>=2.1.0->tensorflow)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/f7/f8/2da482a6165ef3f28d52faf8c2ca31628129a84a294033eb399ef500e265/google_auth-1.11.3-py2.py3-none-any.whl (76kB)\n",
      "\u001b[K     |████████████████████████████████| 81kB 25.2MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting google-auth-oauthlib<0.5,>=0.4.1 (from tensorboard<2.2.0,>=2.1.0->tensorflow)\n",
      "  Downloading https://files.pythonhosted.org/packages/7b/b8/88def36e74bee9fce511c9519571f4e485e890093ab7442284f4ffaef60b/google_auth_oauthlib-0.4.1-py2.py3-none-any.whl\n",
      "Requirement already satisfied: requests<3,>=2.21.0 in /Users/yaramohajerani/opt/anaconda3/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow) (2.22.0)\n",
      "Requirement already satisfied: h5py in /Users/yaramohajerani/opt/anaconda3/lib/python3.7/site-packages (from keras-applications>=1.0.8->tensorflow) (2.9.0)\n",
      "Collecting rsa<4.1,>=3.1.4 (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow)\n",
      "  Downloading https://files.pythonhosted.org/packages/02/e5/38518af393f7c214357079ce67a317307936896e961e35450b70fad2a9cf/rsa-4.0-py2.py3-none-any.whl\n",
      "Collecting pyasn1-modules>=0.2.1 (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/95/de/214830a981892a3e286c3794f41ae67a4495df1108c3da8a9f62159b9a9d/pyasn1_modules-0.2.8-py2.py3-none-any.whl (155kB)\n",
      "\u001b[K     |████████████████████████████████| 163kB 62.5MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting cachetools<5.0,>=2.0.0 (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow)\n",
      "  Downloading https://files.pythonhosted.org/packages/08/6a/abf83cb951617793fd49c98cb9456860f5df66ff89883c8660aa0672d425/cachetools-4.0.0-py3-none-any.whl\n",
      "Collecting requests-oauthlib>=0.7.0 (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.2.0,>=2.1.0->tensorflow)\n",
      "  Downloading https://files.pythonhosted.org/packages/a3/12/b92740d845ab62ea4edf04d2f4164d82532b5a0b03836d4d4e71c6f3d379/requests_oauthlib-1.3.0-py2.py3-none-any.whl\n",
      "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /Users/yaramohajerani/opt/anaconda3/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow) (3.0.4)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /Users/yaramohajerani/opt/anaconda3/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow) (2019.9.11)\n",
      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /Users/yaramohajerani/opt/anaconda3/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow) (1.24.2)\n",
      "Requirement already satisfied: idna<2.9,>=2.5 in /Users/yaramohajerani/opt/anaconda3/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow) (2.8)\n",
      "Collecting pyasn1>=0.1.3 (from rsa<4.1,>=3.1.4->google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/62/1e/a94a8d635fa3ce4cfc7f506003548d0a2447ae76fd5ca53932970fe3053f/pyasn1-0.4.8-py2.py3-none-any.whl (77kB)\n",
      "\u001b[K     |████████████████████████████████| 81kB 22.9MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting oauthlib>=3.0.0 (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.2.0,>=2.1.0->tensorflow)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/05/57/ce2e7a8fa7c0afb54a0581b14a65b56e62b5759dbc98e80627142b8a3704/oauthlib-3.1.0-py2.py3-none-any.whl (147kB)\n",
      "\u001b[K     |████████████████████████████████| 153kB 50.1MB/s eta 0:00:01\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?25hBuilding wheels for collected packages: termcolor, gast, absl-py\n",
      "  Building wheel for termcolor (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for termcolor: filename=termcolor-1.1.0-cp37-none-any.whl size=4832 sha256=feb39afa7dc1c0646e52ef92f492a8f372b4e3e43733b9e4a2988fa36282ee4a\n",
      "  Stored in directory: /Users/yaramohajerani/Library/Caches/pip/wheels/7c/06/54/bc84598ba1daf8f970247f550b175aaaee85f68b4b0c5ab2c6\n",
      "  Building wheel for gast (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for gast: filename=gast-0.2.2-cp37-none-any.whl size=7540 sha256=f1d7c5814ea0e9b24daf6f5a39ce17664e51045a9f770fea1dde796547f05d58\n",
      "  Stored in directory: /Users/yaramohajerani/Library/Caches/pip/wheels/5c/2e/7e/a1d4d4fcebe6c381f378ce7743a3ced3699feb89bcfbdadadd\n",
      "  Building wheel for absl-py (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for absl-py: filename=absl_py-0.9.0-cp37-none-any.whl size=121932 sha256=7c0bf2ad8c8f7b9c3af27466b4f8ffe3fe42f27223d1798dd950d639b9fac8fc\n",
      "  Stored in directory: /Users/yaramohajerani/Library/Caches/pip/wheels/8e/28/49/fad4e7f0b9a1227708cbbee4487ac8558a7334849cb81c813d\n",
      "Successfully built termcolor gast absl-py\n",
      "Installing collected packages: scipy, markdown, pyasn1, rsa, pyasn1-modules, cachetools, google-auth, absl-py, protobuf, oauthlib, requests-oauthlib, google-auth-oauthlib, grpcio, tensorboard, termcolor, gast, tensorflow-estimator, opt-einsum, astor, google-pasta, tensorflow\n",
      "  Found existing installation: scipy 1.3.1\n",
      "    Uninstalling scipy-1.3.1:\n",
      "      Successfully uninstalled scipy-1.3.1\n",
      "Successfully installed absl-py-0.9.0 astor-0.8.1 cachetools-4.0.0 gast-0.2.2 google-auth-1.11.3 google-auth-oauthlib-0.4.1 google-pasta-0.2.0 grpcio-1.27.2 markdown-3.2.1 oauthlib-3.1.0 opt-einsum-3.2.0 protobuf-3.11.3 pyasn1-0.4.8 pyasn1-modules-0.2.8 requests-oauthlib-1.3.0 rsa-4.0 scipy-1.4.1 tensorboard-2.1.1 tensorflow-2.1.0 tensorflow-estimator-2.1.0 termcolor-1.1.0\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "pip install tensorflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) Get data and develop neural network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "#--  import required packages\n",
    "import keras\n",
    "from keras.datasets import mnist\n",
    "from keras.layers import Dense, Activation\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://s3.amazonaws.com/img-datasets/mnist.npz\n",
      "11493376/11490434 [==============================] - 1s 0us/step\n"
     ]
    }
   ],
   "source": [
    "#-- Get built-in MNIST data from keras\n",
    "#-- \"Dataset of 60,000 28x28 grayscale images of the 10 digits, along with a test set of 10,000 images.\"\"\n",
    "#-- https://keras.io/datasets/#mnist-database-of-handwritten-digits\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4sAAABDCAYAAAA1QJAnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO2deZzN9f7Hn58rS7aQpShRyhapW3aXNtx0m4SMLa6QYlxJliIi+pmYi6ToChFGmxDCyNK1JLKUZXBRYmRorGNMfH9/fOfzcc7s2/l+v4f38/GYh3zPmXNefX2+n+W9KsuyEARBEARBEARBEARf/uK2AEEQBEEQBEEQBMF7yGFREARBEARBEARBSIEcFgVBEARBEARBEIQUyGFREARBEARBEARBSIEcFgVBEARBEARBEIQUyGFREARBEARBEARBSIEcFgVBEARBEARBEIQUBPywqJRarZS6qJQ6l/SzN9DfmVOUUiWUUl8qpc4rpQ4rpdq7rSmzKKXuTrrfs93WkhFKqd5KqR+UUglKqRlu68kMSqmqSqlVSqnTSqn9SqmWbmtKD6VUfqXUtKRxfFYp9aNS6u9u68qIIB0bs5VSx5RSZ5RS0Uqpbm5ryohgvM+aIJvrgnFsBOPaHXT3GUApFaqU2p205ziglGrktqb0CLZ5w2cM65/LSql33daVHkG8dldQSi1RSv2hlIpRSk1SSt3gtq6MCMJn0NG9qFOexd6WZRVO+qns0HfmhPeAS0AZoAPwvlKquruSMs17wGa3RWSSo8BbwEduC8kMSRPeV8BioATQA5itlLrHVWHpcwPwK9AYuAkYCsxXSlVwUVNmCKqxkcTbQAXLsooCTwFvKaX+6rKmjAjG+6wJprkuGMcGBN/aHXT3WSn1ODAG+CdQBPgb8D9XRWVMUM0bPmO4MPa+Lh741GVZGRGsa/dk4HfgVqAWtv6XXFWUAcH2DLqxF5Uw1GQopQoBrYChlmWdsyzrO2Ah0MldZRmjlAoF4oAot7VkBsuyvrAsawFw0m0tmaQKUBb4t2VZly3LWgX8Fw+PDcuyzluWNdyyrEOWZV2xLGsxcBDw9AYqCMcGlmX9bFlWgv5r0s9dLkrKkGC8zxCUc13QjY1gJEjv85vACMuyNibN0b9ZlvWb26LSI1jnjSRaYx9m1rktJD2Cde0GKgLzLcu6aFlWDLAM8LqzJdieQcf3ok4dFt9WSsUqpf6rlGri0Hdml3uAy5ZlRftc247HB7tSqigwAnjFbS3XMCqNa/c6LSS7KKXKYI/xn93Wci2ilJqslLoA7AGOAUtclnTNEaxzXZCOjWBau4Hgus9KqTzAg0CppFCyI0lheze6re0apjPwsWVZlttCskIQrd0TgFClVEGlVDng79gHRk8SpM+g43tRJw6LA4E7gXLAVGCRUsrLlr7CwOlk105ju6a9zEhgmmVZv7ot5BpmD7ZF8lWlVF6lVFPsEIuC7srKHEqpvMAnwEzLsva4redaxLKsl7DnikbAF0BC+r8hZIOgnOuCcGwE29oNBN19LgPkxfZ2NcIO27sfGOKmqGsVpVR57DV7pttaskKQrd1rsJ0rZ4AjwA/AAlcVpU8wPoOO70UDfli0LGuTZVlnLctKsCxrJrar9IlAf28OOAcUTXatKHDWBS2ZQilVC3gM+LfbWq5lLMtKBJ4GWgAx2J6N+dgToqdRSv0FmIWdi9vbZTnXNElhId8BtwEvuq3nWiLY57pgGhtBuHYbgug+xyf9+a5lWccsy4oFIgiS+xyEPAd8Z1nWQbeFZJZgWruTtH6DbaQpBJQEimPnA3qVoHsG3diLulGhyCJ1F6pXiAZuUErdbVnWvqRr9+Ft138ToALwi1IKbO9oHqVUNcuyHnBR1zWHZVk7sC04ACil1uNxK6WyB8U0bAvaE0kTjRB4bsD7+VLBRhOujbkuGMeG19fu1PD0fbYs6w+l1BHseysEnueA/3NbRGYJwrW7BHA7MCkpdzhBKTUduxjSAFeVpUGwPoNO70UD6llUShVTSjVTShVQSt2glOqAXWXom0B+b06wLOs8tlVkhFKqkFKqARCCbdnxKlOxF8RaST8fAF8DzdwUlRFJY6IAkAd7w1fA6yWWlVI1k3QWVEr1x674NcNlWRnxPlAV+IdlWfEZvdkLBNvYUEqVTiq9XVgplUcp1QxoB6xyW1t6BNt9JgjnumAcG8G4dgfjfU5iOhCWpL840Be7yqFnCcJ5A6VUfeyQaq9XQfUlqNbuJK/cQeDFpDFSDDtHdLu7yjIkGJ9BZ/eilmUF7AcohV3a/Cx25bqNwOOB/M5c0l0CO8b6PPAL0N5tTVnUPxyY7baOTOq0kv0Md1tXBprfAf7ADldeClRyW1MGeu9Iuq8XkzTrnw5ua7uWxkbSXLcmaZ47A+wEurut61q7z2no9/RcF4xjIxjX7mC8z0m682K3G4jDDimbCBRwW1cGmoNu3gCmALPc1pEFvcG6dtcCViftk2KxD+el3daVgeZgfAYd3YuqpC8VBEEQBEEQBEEQBIP0WRQEQRAEQRAEQRBSIIdFQRAEQRAEQRAEIQVyWBQEQRAEQRAEQRBSIIdFQRAEQRAEQRAEIQUZlTr2avWb9Ho9iebcQzQ7g2h2BtHsDKLZGUSzM4hmZxDNziCaneGa0iyeRUEQBEEQBEEQBCEFclgUBEEQBEEQBEEQUiCHRUEQBEEQBI8QHR1NxYoVueOOO7jjjjvcliMIwnWOHBYFQRAEQRBcJiwsjLCwMBo2bMgvv/zC/fffz/333++2LEG47vjf//5H27ZtyZcvH/ny5WPPnj1uS3KVjArcXPds2bIFgEmTJjFz5kw6d+4M2JP6Aw884KY0Qbgm+de//gXAxIkTuffee1m8eDGAWNiFbPPII4+Y/161apVj37tr1y4zfqdMmULt2rX9Nv99+/YlX758jukRvMnx48dp2bIlGzduBEApRY0aNZg2bZrLygTh+mL9+vUANG/enJIlS9KrVy8AypQp46Ys1xHPoiAIgiAIgiAIgpACZVnpVnDNUXnXy5cvA3D69Gm/65MmTeLChQvs3bsXgPfee4/+/fszd+5cAAoUKMCgQYMAGDZsWKq6A6XZl23btvHwww8DcObMGb/XbrrpJk6dOpWVj/NEGd2oqCg6dOgAwJo1a6hcuXJ6b3dN81tvvQXAG2+8gWVZrF69GoDGjRtn9KueuM9ZxDHNZ8+eBeDcuXN8/fXX/P777wC88sor5M+fPysfFRDNhw4dMh77uLg4lFIsWbIEgGbNmmX3YzUB0RwdHc2lS5cAWLduHS+99BJKpf1VTz/9NPPmzQPIjFcpYGMjMTERsC2pgwcPNhbVXMAzz+DLL78MwAcffMBzzz0H2B6+VMh1zVOmTKF///6cO3cuzfdERUX5eT2ziGfucxbItOZz584RGRkJQP78+dm6dauZv2bPnm3W5nLlyqX4oFtuuYWQkBAAHnzwQcc0Z5Xo6GgA+vfvz9dff43ej40ZM4YHH3zQ/D9mg4BotiyLdu3aAbBkyRJ27drFbbfdlt2PS841PZ6zwqxZs/jmm28A2L59u9krA9StW5dFixZx0003ZffjXb/P58+fp0mTJvz222+AvQZVqFAhvV9xRPPixYtp06YNAD179mTUqFEULFgwux/n+n3OBmlqzrUw1F9++cVsmNavX893331HXFwcAJ999lmqv3P77bcDdkjnl19+SZEiRQC47777MnMoCCjff/89rVq1MgddpRRFixY1m7vY2Fg2bNjAX//6VyBTm74ssXbtWgBOnjxJy5Ytc+1zN2/enBuLZ0CZMWMG//d//wdAnjx5uHz5crqbbyFjDh48SHh4OBs2bABg586dfq/HxMQwceJEN6T5UapUKfPsf/XVVy6rSZuffvoJgJkzZ/Lpp59y5coVAH777TeUUumO16+++oqePXsCMH78eIoWLRp4wamg57YmTZpwyy23EBMTA9gb7WuBQYMG8cEHHwCQN29eHn30UUe/v02bNrzxxhvpHhZbtWplDkRNmzZ1SlpQMGLECN555500X1+6dGm6vz969GgAqlevTmhoqDnkVKxYMfdE5pCTJ08C8PXXX/tdv+2223JyUAwY8fHxfPfdd4BteFy2bBndunVzWdW1QWxsLADdunVj4cKFFCtWDID69etzxx13sGbNGsA2SNatW5fdu3e7pjWzHD16FIATJ04AULx4cQC+/fZbfvjhB6pUqQLAzTff7I5AH/bt28ezzz5r9h/jxo3jL3+R4EtNjg+LP/74I2DnhCT3IKZHnjx5jPeoUKFCdOjQgbJlywL2gMrA4xUQLly4wNatWwHo2LGjGeiau+++mwEDBgDQtm1bGjRoYP4fXnvttVzVoj1p+/bty7XD4pUrVzh48CC//PILABl4lV3j8OHDJCQkuC0jBZs2bQJsq9/atWvNgQHsiUWP33Xr1tGpUyfq1Knjik7Nnj17GD9+PGBb4uPj482/efny5SlSpAi7du0CYP78+bz00ktm8naLQoUKBUVuon7ek2/yMsvMmTMB6Nq1Kw0bNsw1XdklJibmmjssbty40RgwGzZsyLPPPuvo95coUYI333yTfv36AfZGu3z58mb+Bdt7vmzZMiB4D4uHDx8mPj4egLlz5/L++++b11q0aMH06dOz9bmff/55imslS5YEoEaNGile03PXnj17iIuLM3uTnTt3snPnTmrWrAl457AYHR1N+/btgatr8ZdffglgvKJeo2DBgtxzzz2AbRjTkSnBxrhx4wC4dOkSu3fvZvbs2ea1KlWqmHXRSXTkzKFDhxg4cCCvvvoqYM8jgCmwUrt2baKjoxkxYgRgR195gZ07d/Luu+8C9pwAVz3n+u86YlAfdPWeSc/TbnDx4kUAunfvTs2aNZk/fz5AUBwUT506RWRkpDGMaU9tIM4l3r8bgiAIgiAIgiAIguPk2LOovQAlS5ZM17NYp04dPxd0vnz56NSpU06/Pld54YUXmDNnTpqvb9myxYQUNW7cmNWrV6cI58sttOehfv36ufaZx44dY+rUqea+u+1FSo2VK1f6hUNWqVKFxYsXu16JKjIy0lTpPHHiBJZl0aRJE8AOH+nfv795r2VZxMbGmrw0J9HP4MCBA4mMjEyRa6utwt988w2XLl0yY+DEiRMmDMZN4uLi2L59u9syMuTxxx8HrnoWS5cuDcDzzz/PlStX/KyS69evNyFEQs7Q4fmjRo1i7ty5xuqenLlz57Jz504qVaoEwNixYx3T6EvPnj1NKOz27dtTDTnu3bu307JyzMqVKwH44osvmDt3rkk5SR5+rat7Zofly5ebXC0daaTzh2699dZ0f/fs2bPG+6i9GosWLQLgySefzLam3GTWrFnGy9yiRQs++OCDVPMvvYauDvntt98GTTsBPf/u3LmTtWvXGg+uTh/wHbf79++natWqAI6Feq5YscJ4wtu2bcvbb7+d4j16re7bty8jR440HnuveBa//fZb/vOf//hd03UQOnXqRFRUlEkv0vzzn/8E3A1DHTp0KGBHju3bt8+1tJCsoNOJ+vXrx6ZNm8z41X/q/6d9+/ZlO7IjOTk+LOrF+p133jGT8f3330+fPn3Me2rVqsXKlSspVKgQYOf7eCE/ypctW7awePFiv9DMJk2amIWlf//+lC1b1pQ9L168ON9++23AQjn1JJab6NyCu+++O9c/O6foPIguXbr4HXBeffVV18IS//zzT8DO8+zevTvnz58HbEPB0KFDTfhgQkICzz77rElIh1wpqpAt9CL44YcfpnitUqVKrFixArDzhfft2+eotsxw4cIFs7nTbN68GbAXS6+EqL744ouAXawG7Jw4SD2E88yZM9x7773A1TAR/XsPPfRQwLVmFh1K6GV69OgB2OFNu3btSjOEd9SoUZw6dcpsXu677z7HNCZnyJAhRtO2bdtSvO7FkPu0eP755/npp5/4/vvv/a7rDVaHDh3M3Ne+fXsKFCiQ7e+66667uOuuu7L1u4sWLfKbRwoUKOCp3Lp69eqxbds2U9QjIiIiKA6KYIdBaubPn8+YMWOAjA/wTnHs2DGTn/q///0PuGpEPXfuHJZlmTGqW6P5cvnyZS5cuOCQWpvExESzLwsNDU33va1bt2bkyJEmfPLMmTOuH3CGDx9OeHi4+XuXLl0oVaqUMaKXKlWKbdu2mVDbEydOULp0aVq3bu2KXk1CQoIJQW7SpEluFmsKGLGxsWYd3LVrF6VLlzb7iZCQED7++GMTSqtTMXKjpoqEoQqCIAiCIAiCIAgpyLVqqE8//bQpAV6kSBF27NhhrLr9+/c3XkWAe++9l6lTp+bWV+cIbel97LHHOHPmjHHjPvHEE8ydO9cUmhk1ahTdunWjVKlSgG2pVkqZMLStW7eakv85ZceOHRw/fjxXPssXHSqkQ+i8hA671UWFdIinLnfvBtri9PzzzwNXC1BERkb6WfIiIyP9vIq33347nTt3dlDpVbRFSaMt17Vr12bMmDGmAjHgyRCismXLmtAU3TZH/1msWDHPhOzdcIM9dfrez7T45ptv+OOPP/yu6d/LYruSgKKt7PXq1XNZSdrceOONgB1uoy3rvuj5/JdffknzPU6jrecNGzakadOmKVIXtOcxtYIuXuDkyZMMHjwYgI8++ogSJUoYz8ygQYO49957zb9L+fLlXdOpi2T06dPHrCea9evXm6ggN9EVnnXomC66pO9fsJGQkMDChQsBO43HTXRodPfu3f2KSCVn9+7dplBSbGwsR48eNWvOr7/+CkC1atUCrNafRx55xIShZtSqQa8ZuiDZnDlzTHVttzh//jzx8fFmvzFq1Cg/T/P+/fsZPXq0KYhUqFAhhg0blqPIg9wgPDzcpJaNGjXKVS2Z5amnnjIFmJo1a2Zai2kqVapknoUjR46we/fuXImsybXDIuC3gfbtAfOf//yH0NBQz1UXio6ONq7z06dPU6pUKTPAO3fuTOHChU0Yalp5DjpcYezYsenmO2aFJUuW5GpImD54Hjp0CEi9L5WbxMbGMm3aNMCuklusWDGzgXKLIUOGmApTSil69eplKkwlD/lIPslMnDjRGBWcRhtopk6dStOmTU3Ols6p8yUQBoncQMfbp9FjNaiYN28eU6dOTRHWpCvZuYk+8BYrVoy4uDgOHDjgsqL0GTp0qKlAXLVq1RQL4Pnz501I3Pnz56lbt67rYU5w1ei0Y8eOVHPcGzVq5LSkLDFy5Egzr/Tp04dRo0ZRuHBhl1X5s2rVKnOfdY6ODr2aOHGiyUFzk7i4OJNzq9F1HFILf5swYYLfoUdX8PQablay9EXv5ZIfFPPnz29eq1Onjl+l/ZtvvpkJEyaYQyLYBtZZs2Y5oPgqWTk03XnnnVSvXp2ff/4ZuFpx1E1at27N0qVLzSFm0KBBTJ482YT/9uvXj8WLF5u0tSFDhvDSSy+5plezfPlyGjRoAJBrzp5A42tYyqhqcpEiRYxhJKfk6mHRl+HDhxtL9erVq1m5cqVnSoPrHBHdCBfsA8DHH39sLKZZPaz5TjY5xbcBa/Xq1XP8eTpuPCYmhsqVK5t+ll7g0KFDPPPMM37XwsLCctKoOkfoTfzo0aONBa9Zs2aMGTPG7yG9ePEiy5cvB+wCCpZlmUOOm2XPdSnq4cOHZ/jeXGzCHhC82tolI2bPnm0S+Q8cOJBiM1WrVi2T4+gmuo9Xo0aNTL65V/n111/58MMPzQH3vffeS2GQ6devn/GslytXzvXxvWfPHlq2bMn+/fuBqznQyXnqqaeclJUhFy5cMIfujz/+mAkTJpief82aNXPdG5Cc77//nmbNmqW4vzpK6PbbbydPnjxuSPMjT548pjWXntv+9re/+b0nIiICsLVPnDjRL+8yIiKCI0eOAN4z+LrN8uXLUy2mVL58eWbNmpVueyJ9TzUhISG5tsEOBHnz5vXE+uFLrVq1qFevnjksRkVFsWLFCl5++WXgapEpvS8JCwtzRacv69atY+PGjezYsSPV11evXk3JkiVNvQGvYFmWmT+KFy/OxYsXzRozc+ZMtmzZYmonzJkzJ9fmCm+5+gRBEARBEARBEARPEDDPYqFChUxFxgceeIDu3bsb6+SDDz5Ir169UpTZdgpt3fNtqP3VV1/RuHFjV/SkR3aqJepqosuWLWP27NnGAwa2+197FLzAsmXL/EKzHn30UdOiwmni4uKYPHkyYFt2deWuBQsW+L1v//79dOjQgR9++MFca9OmDQMGDHBObBaZOHEi58+fNxYppZQJ6QNo0KCB53LVlFKuzREZoUO6Z82aZfIDNOvWrUuhW4cujxkzhieeeCJoc5ScRs8NzzzzDCdOnDBVtpPP1WPHjmXGjBnm76+//rpjGtNi9+7dHDx4ME2Poubf//43gGlo7TZvvfWW8Yy3bduWpk2bes6b6EtkZGSq91hHELVo0YKHHnqIf/zjH4BdX0G31XCSNWvWmDBUpRR33HGHX8uAbdu2margOrdRh/uWK1eOvXv3mrDqefPmeaYytBcYN26cqVYOmNDCYcOGpepV1DnkS5cu9QsNbtCgAS1atAiw2pyRkJDgl4vtdiVUsEN9fSPWjh49yjPPPOO33+jWrZup2ukFPvnkE6pWrcqdd95prs2YMYN+/foB9hgpUKAA77zzDuCdFke7du0y+4uIiAjGjRvntxeNjIwMSPpFwA6LgCl7PWPGDP75z3/y8ccfA3Zoy/nz503xEqdLLuvB4NsrLzsHRd8wuUCFzJ06dSrFNd2H7sqVK0RFRZkwikuXLvHJJ5+Yths33ngjderUMeGUiYmJrrV0SI4+gA0aNAi4mrczc+ZMv3xXJ7l06RInTpwwf9ftXX7//XemT59uFvCff/6Zs2fPmgf2L3/5Cx07dvQr4uQ2Fy5c4OeffzZhtdow4jt5w9Ww1enTp3siVCsY2LlzpwkdTK+Qgi863EyXvPYiJ0+edFsCcDVcc/bs2XTt2hWwx61SyvSXGj16NK+88oqZHz/99FMsyzKFpdwutgHQsmVLwsPDGThwIECaxXZ0US+v4NvjrV27dp4+KAK0atWK3bt3mw2T7xyu2bx5s2nBM3z4cPr27Wv+XVLL585Nzp49C8DBgwfNtbJly9KpUyfTLkHXT9DrYqlSpXj88cd55ZVXANsA/PDDD5sidYI/PXr0MP/uxYoVM/UjUmtlBJjep7o2gg41nD9/fpq/4xUOHTrkV5yuefPmfq/HxsaaPeKGDRto06aNX55moNDFbVKjRYsW9O/fP1MF4Zzio48+Ys6cOWZ/fOnSJd58801TfFMXj+nSpQtgF45Jfq/doESJEsYhtHnzZrM2gu2kC1RxpoAeFjUtW7akUqVKZuJbuXIlgwcPNnHMr7/+umMx+IsXLzYV85RSOcoX8fV81KpVK1f0gX3I05/7wgsvmEIrGj0RWJZF3rx5TfWsqlWr0rVrV/76178CdkXRMmXKmOT5+Ph409jVTVLLU9TWnTJlyrghCbALIuiNw++//24mv+ReonLlylG0aFGzyStZsqSxWrtJYmKiqajWqlUrjh49asZG2bJlqV+/PsuWLQMwVtjLly8DdnPtf/3rX7nSj+d6IjUjUWrXdE7gkiVLeOKJJwKuKzvoqoZuM2/ePMCuQuz77N19991mw79582YWLlxo+lYePXqU0qVL89FHHzkvOB369OljDgR6o68Pw7179/brKesVateube5z7969ufHGGz1ZQVtTv359lixZYgw3sbGxHD9+nC+++AKAadOm+T2TV65cISIiwkQYRUVFBbT4nvYW9u3b11zr0aMHb7zxhikypusnaC9RmzZtGDdunOmF27NnT4oWLcqjjz4KIF7FZLRq1YpWrVpl6r2LFi3yKzCWN29eY1zy6kExISHBOAX++9//+r3Ws2dPHnjgAbP2nzp1yjwLRYsWZf/+/X5RF4Hg8uXLrFu3LsXapwtDeiknXkdUJSYmmhx4sCMOmzdv7ueVa9u2rXl+3377bU8cFnft2mXyc48cOWIqKoMdgROow6LkLAqCIAiCIAiCIAgpcMSzCFCjRg1TqW7RokV06dLFhALs27ePFStWOKIjPj7eVCcsXbo0bdu2zdLv6zwIXdVJW/p0jkduMHnyZGM5TK2in+5lFRISQrVq1ahbt26anzV16lTT28Y3NttNxowZkyLkUYejukmxYsVMGNCTTz5pwvIqVapESEiICUcoUaIEoaGhxrMYGhrqil6NHs/Lli2jZcuW5vrw4cNNnnDDhg05deqUqTKrc8H02Bg0aBDly5c3OQVe6P+X3Eq5du1aT+QN1KhRw/RfnTVrFs2bN08zVG/atGkmnNmrPPzww56x/EZGRpqeZ/ny5TP51XPmzKF48eImhWDNmjUmBAds739sbKwJc1q9erVJg3Cbv//9735/15r379/PiBEjTKTL4cOHXfEYbdq0yfQgzJcvH0uXLjVjdsSIEbRu3dpYsr3QgiIt9Lqo/9T3vXHjxkyaNIlNmzb5vV8/w2PHjg1ovnlq1RbfeOMNADNfa2061aFx48Zs2LDBL9+ub9++nmyfUbNmTbclZImQkBC/iIWJEye6nh4QHx9v1uItW7awadMmVq1a5fe6bpWRnJ9//tm0qADo2rWrybu8+eabqVixYgCV24SGhvL555+niMLyYs0B35ZhvuG51atXN+3RfHnxxRcBPFUVVe/5k7dieu211wL2nY4dFuFqqfZOnTrRrVs3EhMTAXsTuHr1apM/6BQFChTIUr5kQkKCGUzh4eHcfvvtJrQ2t/tO6XyKnBIVFWX+2ws9x7Zt2+bXwB7s0vFOxNRnhjp16gCp571o1q5dy5o1a8xE6OYhPDEx0fQj1L2kwN4ohYWFmWfuxIkTPPHEE2bjkj9/fgYMGGAmm6+++or27dubcLMBAwaYHmCAKw2tkxe4+fzzz9m1a5fjDZNTQ2/qM+oHOnz4cM8fFvXmWhsd3Dq0AEyZMsUc+IYMGWJyFjWTJk0C7DA+nb+ouXLlijGOeOWgmBr6PutQOB367WTO8LFjx8yG8tdffzVFdjp27EiJEiWMUWbEiBGcPXvWFAQJRjp27EhoaCiPPfYYYBsafNFl5wOFDriE9PIAAAwwSURBVD+2LMuvwMe2bdtMoSzLsoiIiDC1E6Kjo2nfvr0xLERERPiFsXoJLz9ryXnttddSGCHdKmwYHx9vnA4LFy70y0P05aabbqJw4cKmXYbeN3fv3h24GobqNEePHjVh/5999hlKKZMCVbNmTaZPn24OwF7Ft79pWi3lUuuB6hV++uknx1qMOXZY3LFjB5999hlg55voAQ9QrVq1FP2GnCAr+Yrbtm0jPDycyMhIwLZO6ZyIYMELlaiaNm3qt/GoU6cOM2fOdFFR1omPj/c7yLjlWbx8+TJDhw411boKFy5silO0a9eOYsWKmdyjsLAwtm7dyj333APA+++/z8MPP2xyptavX88nn3xi8tZ8c5TKly/vV5zBKXr27MmUKVP8rk2dOpXx48c7riW7JDeMeBGdt6EXHR094QYhISEmnzm1YgixsbEAxsqu8xu11dfLC7smuYHh+eefB5zV/sADDxhvRHh4OB07dvR73fcZe/zxxz1lVc8ON9xwg9lQJz8s6jkx0KTmZdEGAqUUO3bsMIabixcvUrFiRZMv5VbRt2sFbaD58ccf/dbuCRMmmJxip3n66adNpfoCBQqY/L6KFSsSEhJionsqVKjAbbfdZupN7N27lzvvvNP05MxtR0VmiYqKMh5ygFGjRhkj04IFC5g+fbonDLvJyerhSs8XXqg6mxxd30Q72gJZc0JyFgVBEARBEARBEIQUBNSzuHfvXsDuH/XFF18QExPj/+VJFu1bb701oNXIfLEsy1gWFixYwIQJE9J9v7bejBw5ktOnTxsLrG4DImSN2NhYv3CrXr16uWYZyy66/6LbTJ06lXfeece07JgyZQpNmzYFYOPGjUyfPp0lS5YAtjd02LBhJh9Me220tax58+Y0b96cuXPnAnYPIo0OUXMaL+VHJSYmGi/ho48+mmGfRB2e49XQMV9CQkKoUqWKCYMaP3686TfqNOn1WD19+rTJez99+jSVKlXyqwTnFidPnjTPVWhoKO3bt0/zvceOHTOl2TXJK0M7QZ8+fRg5ciRgRx2EhYWZ1+655x6io6MB26vx9ttve86qfuzYMdPHuUqVKhmOg8uXL5sq4hod1qdTDwKFjmAKDw83OYkbNmxg+/btpq0G2G2j9N6kVKlSDBs2zLEq8TnBzUiEzHDhwgVmz54NYDx5+hnt2LGjY3vP5CxfvtxUXP/iiy/STPX4888/GThwoKmGWqZMGT799FPX9k0611f3vAW7Dsljjz1m9vg6xD69dhpukZU8ysTERN5//33ATp/zCrt37wbsmgilS5fmpZdeAgJ7vwNyWIyJiWHOnDkmv0TH5fvy0EMPmcbJOWlfkVV8QxBiYmLMgO/atSs333yzSeSfNWsW27dv59dffwXsHKXmzZubf5RgZN++fa41XtebKcuyTLsGsMueBxteCS3UE7IuxR8eHm5yIHTJdc2bb77J4MGDM8yLateund+fbhIWFsa7777rl1M0YcIEs7F1Kldm3bp1jB492mw0Dh06lGa/qFOnTrFkyRKTy6xblOgWJhkdMt2iWbNmpmCTNpB5jcmTJ5uFu0yZMn4FINwkLCzMFAiKjo42G/xy5cpRqVIltmzZYl4LDw/3a5fRr18/0+vUSQYPHmwOS1u3bvXLbf/jjz9MPuO4ceOoVKmS4/rSQm9GmzdvbvKvM+o9ePz4cSIiIlKMF22M0j1+A4UODStUqJCZDxo0aJDqptW3dYZXW+wkRxskfQ0OXuHs2bN0796dTz/91FwbP368CZd066Co0TUFatSokeI13Ze1TZs2LF682BRRmzdvnis5ihq9DsbFxZnwxyeffJLExEQWL14M2MY8y7IoWbKkWzLTRIfG3nrrrcyePdsUsElOYmIiPXv2NOcXrziITp8+bVp4HDlyhPDwcEfqkeTaYfH48eMmj6R3796pJutqC96AAQMICQlx/UH9888/ee+99wA7Qfemm24yFlWNPsw88sgjfr15gpErV6648r3btm0z1W6VUuTPn98cut3sq5hdDhw44LYEwO4J9fvvvxvLrq/lvEWLFvztb38zeaoVKlRwtIBGblG9enXX73dYWJhf1bHw8PA0k+FXrFjBli1b/DaCTZo0MeNdF2DxIlqzF3ttHj58mA8//NCsGT169PBMfmJYWJjJ6d24caPZQFWoUIGqVauavDNfLxLYHrERI0a41vS+f//+rnxvTtCeet8KowcPHqRy5cp+hpj4+HhT8CsiIiJFP8siRYo4VnhKF/2YM2eOMcRo74ymc+fO1KxZ03iX3Cq6khn0ml29evU0K3R6hSNHjvgdFCtVquTnEXOTypUrm0rIPXr0MNXX77vvPu68804zfvfu3UvdunVNtIcbxeZ80XOwr+MlMTGRBQsWmHtbvHhxunfv7knnii5q+dprr5nq2gAdOnTgwIEDZm4ZPXo0BQoUMHtXrxx8BwwYYLzM7dq1M4bpQCM5i4IgCIIgCIIgCEIKcuRZPHXqFC+88AJge4/S8gA0aNCAV155xeR6uRmKVa9ePWrXrg3A999/b67HxMT49V8pWbIkoaGhGeY0BhMbNmwwvQKdJC4uzu/eli1b1pP9ojJLo0aNHCtXnB5r165lwYIFbN26FbD7hupWA8WLF/ekhyir9OjRw1Ro9QqZyecrXbo0YIfYT5gwwTXvUVbQ1TEXLFjgSh5dejz++OMcPnzY5I28+eabLiu6Sr169Ux4/3PPPWes6YcOHUo1BUO3pNF5J0Lm0X2NdVVysD0t999/vwnpA3vN+fHHH1P9jCJFivDll1867r178sknTcXLYEavK3ofpz0vXgpD1ZFt2pOrK94uW7bMNU3J2bNnD0OHDgXsXp868ktr1OlZERERJuzQC/i2FStVqhRgz89r164112fMmME//vEPx7VlBR2KrL2LvXr1Aq6Ggvfp04chQ4Z4ah+1cuVKZs2aZdJa2rRp49h3Z/mwqJvHhoeHs3nzZuMOTU7BggWNS/r11183RTjc5rbbbjMtL6ZMmWKS/DW6wMKLL77oWkllwdvUqFGDu+++2xhHDhw4YCZNJylSpAidOnXyVOJ1blOtWjWTY7Br1y5XNEyfPp1333033RYvOq+rYMGCNGrUyPTASi0XxYtERkaaA60Xy5136dKFoUOHOprfnhX0pjQhIYFz586Z6z/++KMpGgV2C4SVK1c6ru9aQfdKbNeund99TetgqMmbN68JYW3VqlXAi9pcD9SqVYsffvjBb7x7BZ0ypI0K+iDrVv/YtND7z+T7UC/jW3hOh/haluXXn1U/p16nd+/eRrOX0UZHXchL70VCQkIc0yBhqIIgCIIgCIIgCEIKVAbhdCleHDRoEIBJvtVUq1bNuJ3z5MlD//79/cJCcpn0at+6Hx+YOq5onjFjhqlE2qNHjxRNzjMgVzTHxMTQtm1bwK4sWbFixUAWLXHkPs+YMcM0027cuDGTJk3KiUdGxrMzZFtzQkICM2bMAOym6qdOnQLsxspNmzY1Fr5bbrklV4T64Mh9Dg0NNWGRCxcuzKkF/roaGy5y3WpOSEjgyy+/BGDVqlXcc889fuHquoE52MXpKleunJPCINftfU6PQ4cO0a5dOzp37gxAz549c/qRuaL5p59+YvDgwYBdqfWFF14wEWOVK1fOkcBUuO7Gxh9//AHAhx9+aDyiDz74IE899RQvv/xyrghMhevuPmvi4+N59dVXAXj//fdp3bq1Xxh+LpOm5iwfFj3CdTtwHEY0p8GZM2dMSMCKFSto1aoV06dPB8hOyLXcZ2cQzc4gmp1BNDuDaHaGXNE8cOBAxo4dC9hhp0uXLg3EIVFz3d5nh7luNU+ePNmEytavX5+oqCjy58+fU21pIYdFDyCancExzboc++uvv87kyZNNe4VseBjlPjuDaHYG0ewMotkZRLMz5IrmqKgomjZtCtjN7gOc13Xd3meHue406wKczzzzjIli6969e6BbRqWpWXIWBUEQBEEQBEEQhBSIZ9E5RLMziGZnEM3OIJqdQTQ7g2h2BtHsDKLZGUSzM2Q7DFUQBEEQBEEQBEG4DpEwVEEQBEEQBEEQBCEFclgUBEEQBEEQBEEQUiCHRUEQBEEQBEEQBCEFclgUBEEQBEEQBEEQUiCHRUEQBEEQBEEQBCEFclgUBEEQBEEQBEEQUvD/z+MtCzUr1dQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1152x360 with 20 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(1, 20, figsize=(16,5))\n",
    "for i in range(20):\n",
    "    ax[i].imshow(x_train[i],cmap='binary')\n",
    "    ax[i].set_title(y_train[i])\n",
    "    ax[i].axis('off')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#-- Turn the trainign labels (indicating with number each image is), into one-hot encoding\n",
    "#-- e.g. 3 --> [0,0,0,1,0,0,0,0,0,0]\n",
    "onehot_train = keras.utils.to_categorical(y_train, num_classes=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "#-- Make sequential model\n",
    "model = keras.Sequential()\n",
    "#-- \"Dense implements the operation: output = activation(dot(input, kernel) + bias)\"\n",
    "#-- https://keras.io/layers/core/\n",
    "#-- first hidden layer has 64 units, and input is 28*28 which is the flattened input data\n",
    "model.add(Dense(64, activation='relu', input_dim=x_train.shape[1]*x_train.shape[2]))\n",
    "model.add(Dense(32, activation='relu', input_dim=32))\n",
    "model.add(Dense(10, activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is categorical classification with 10 classes (0-9), hence 10 units in the last layer. We compile the model with a `categorical_crossentropy` loss function used to train the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer='adam',     # optimization algorithm used (other examples include scholastic gradient descent, etc)\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy']) # quantity to be minimized"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "60000/60000 [==============================] - 2s 37us/step - loss: 1.6560 - accuracy: 0.7003 0s - loss: 2\n",
      "Epoch 2/20\n",
      "60000/60000 [==============================] - 2s 32us/step - loss: 0.4983 - accuracy: 0.8652\n",
      "Epoch 3/20\n",
      "60000/60000 [==============================] - 2s 33us/step - loss: 0.3102 - accuracy: 0.9179\n",
      "Epoch 4/20\n",
      "60000/60000 [==============================] - 2s 33us/step - loss: 0.2295 - accuracy: 0.9384\n",
      "Epoch 5/20\n",
      "60000/60000 [==============================] - 2s 30us/step - loss: 0.1924 - accuracy: 0.9477\n",
      "Epoch 6/20\n",
      "60000/60000 [==============================] - 2s 35us/step - loss: 0.1638 - accuracy: 0.9553\n",
      "Epoch 7/20\n",
      "60000/60000 [==============================] - 2s 32us/step - loss: 0.1426 - accuracy: 0.9609\n",
      "Epoch 8/20\n",
      "60000/60000 [==============================] - 2s 35us/step - loss: 0.1318 - accuracy: 0.9632\n",
      "Epoch 9/20\n",
      "60000/60000 [==============================] - 2s 41us/step - loss: 0.1219 - accuracy: 0.9673\n",
      "Epoch 10/20\n",
      "60000/60000 [==============================] - 2s 38us/step - loss: 0.1114 - accuracy: 0.9693\n",
      "Epoch 11/20\n",
      "60000/60000 [==============================] - 2s 33us/step - loss: 0.1067 - accuracy: 0.9703\n",
      "Epoch 12/20\n",
      "60000/60000 [==============================] - 3s 42us/step - loss: 0.1005 - accuracy: 0.9725\n",
      "Epoch 13/20\n",
      "60000/60000 [==============================] - 3s 42us/step - loss: 0.0975 - accuracy: 0.9737\n",
      "Epoch 14/20\n",
      "60000/60000 [==============================] - 2s 37us/step - loss: 0.0898 - accuracy: 0.9751\n",
      "Epoch 15/20\n",
      "60000/60000 [==============================] - 2s 36us/step - loss: 0.0879 - accuracy: 0.9766\n",
      "Epoch 16/20\n",
      "60000/60000 [==============================] - 2s 32us/step - loss: 0.0798 - accuracy: 0.9784\n",
      "Epoch 17/20\n",
      "60000/60000 [==============================] - 2s 34us/step - loss: 0.0793 - accuracy: 0.9790\n",
      "Epoch 18/20\n",
      "60000/60000 [==============================] - 2s 36us/step - loss: 0.0745 - accuracy: 0.9796\n",
      "Epoch 19/20\n",
      "60000/60000 [==============================] - 2s 37us/step - loss: 0.0715 - accuracy: 0.9804\n",
      "Epoch 20/20\n",
      "60000/60000 [==============================] - 2s 42us/step - loss: 0.0701 - accuracy: 0.9813\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.callbacks.History at 0x13625f650>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#-- Train model (20 epochs, with batch sizes of 32)\n",
    "#-- batch size is number if data points used in each training iteration\n",
    "#-- epoch is total number of times the whole dataset is used.\n",
    "#-- so total number of iterations is total_size/batch_size * epochs\n",
    "model.fit(x_train.reshape(x_train.shape[0],x_train.shape[1]*x_train.shape[2]),\n",
    "          onehot_train, epochs=20, batch_size=32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) Evaluate model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 0s 16us/step\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.1734668337893905, 0.967199981212616]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#-- turn testing labels to one-hot encoded\n",
    "onehot_test = keras.utils.to_categorical(y_test, num_classes=10)\n",
    "#-- evaluate performance of model\n",
    "#-- Returns the loss value & metrics value, which in this case is accuracy\n",
    "model.evaluate(x=x_test.reshape(x_test.shape[0],x_test.shape[1]*x_test.shape[2]),\n",
    "               y=onehot_test, verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this simple 4-layer model with get $~97\\%$ accuracy!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7.4 64-bit ('anaconda3': virtualenv)",
   "language": "python",
   "name": "python37464bitanaconda3virtualenva1710d598353463ba9e31928f7c4a778"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
